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The  Problem  with  Testing 
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Norman  Hines 
JE  Sverdrup  Naval  Systems  Group 

Testing  is  inefficient  for  the  detection  and  removal  of  requirements  and  design  defects.  As  a  result,  lessons  learned  in 
testing  can  only  help  prevent  defects  in  the  development  of  subsequent  software  and  subsequent  process  improvement. 

Instead  of  testing  out  defects  to  achieve  quality  measures,  quality  should  be  designed  into  software.  Thus  test  devel¬ 
opment  should  parallel  the  development  of  the  software  it  tests. 


Unlike  other  engineering  disciplines, 
software  development  produces 
products  of  undetermined  quality.  Testing 
is  then  used  to  find  defects  to  be  correct¬ 
ed.  Instead  of  testing  to  produce  a  quality 
product,  software  engineers  should  design 
in  quality  [1].  The  purpose  of  testing 
should  not  be  to  identify  defects  inserted 
in  earlier  phases,  but  to  demonstrate,  vali¬ 
date,  and  certify  the  absence  of  defects. 

Beginning  with  the  Industrial 
Revolution,  many  technical  fields  evolved 
into  engineering  fields,  but  sometimes  not 
until  after  considerable  damage  and  loss  of 
life.  In  each  case,  the  less  scientific,  less 
systematic,  and  less  mathematically  rigor¬ 
ous  approaches  resulted  in  designs  of  inef¬ 
ficient  safety,  reliability,  efficiency,  or  cost. 
Furthermore,  while  other  engineering 
practices  characteristically  attempt  to  con¬ 
sciously  prevent  mistakes,  software  engi¬ 
neering  seems  only  to  correct  defects  after 
testing  has  uncovered  them  [2] . 

Many  software  professionals  have 
espoused  the  opinion  that  there  are 
“always  defects  in  software  [3].”  Yet  in  the 
context  of  electrical,  mechanical,  or  civil 
engineering  the  world  has  come  to  expect 
defect-free  circuit  boards,  appliances,  vehi¬ 
cles,  machines,  buildings,  bridges,  etc. 

Follow  the  Basics 

All  models  of  the  software  development 
life  cycle  center  upon  four  phases:  require¬ 
ments  analysis,  design,  implementation, 
and  testing.  The  waterfall  model  requires 
each  phase  to  act  on  the  entire  project. 
Other  models  use  the  same  phases,  but  for 
intermediate  releases  or  individual  soft¬ 
ware  components. 

Software  components  should  not  be 
designed  until  their  requirements  have 
been  identified  and  analyzed.  Software 
components  should  not  be  implemented 
until  they  have  been  designed.  Even  if  a 
software  component  contains  experimen¬ 
tal  features  for  a  prototype,  or  contains 


only  some  of  the  final  system’s  features  as 
an  increment,  that  prototype  or  incremen¬ 
tal  software  component  should  be 
designed  before  it  is  implemented. 

Software  components  cannot  be  tested 
until  after  they  have  been  implemented. 
Defects  in  software  cannot  be  removed 
until  they  have  been  identified.  Defects  are 
often  injected  during  requirements  analy¬ 
sis  or  design,  but  testing  cannot  detect 
them  until  after  implementation.  Testing 
is  therefore  inefficient  for  the  detection  of 
requirements  and  design  defects,  and  thus 
inefficient  for  their  removal. 

Testing  in  the  Life  Cycle 

Burnstein,  et  al.  have  developed  a  Testing 
Maturity  Model  (TMM)  [4]  similar  to  the 
Capability  Maturity  Model®  [5].  The 
TMM  states  that  to  view  testing  as  the 
fourth  phase  of  software  development  is  at 
best  Level  2.  However,  it  is  physically 
impossible  to  test  a  software  component 
until  it  has  been  implemented. 

The  solution  to  this  difference  of 
viewpoint  can  be  found  in  TMM  Level  3, 
which  states  that  one  should  analyze  test 
requirements  at  the  same  time  as  analyzing 
software  requirements,  design  tests  at  the 
same  time  as  designing  software,  and  write 
tests  at  the  same  time  as  implementing 
code.  Thus,  test  development  parallels 
software  development.  Nevertheless,  the 
tests  themselves  can  only  identify  defects 
after  the  fact. 

Furthermore,  testing  can  only  prove 
the  existence  of  defects,  not  their  absence. 
If  testing  finds  few  or  no  defects,  it  is 
either  because  there  are  no  defects,  or 
because  the  testing  is  not  adequate.  If  test¬ 
ing  finds  too  many  defects,  it  may  be  the 
products  fault,  or  the  testing  procedures 
themselves. 

Branch  coverage  testing  cannot  exer¬ 
cise  all  paths  under  all  states  with  all  pos¬ 
sible  data.  Regression  testing  can  only 
exercise  portions  of  the  software,  essential¬ 


ly  sampling  usage  in  the  search  for  defects. 

The  clean-room  methodology  uses 
statistical  quality  certification  and  testing 
based  on  anticipated  usage.  Usage  proba¬ 
bility  distributions  are  developed  to  deter¬ 
mine  the  systems  most  likely  used  most 
often  [6].  However,  clean-room  testing  is 
predicated  upon  mathematical  proof  of 
each  software  product;  testing  is  supposed 
to  confirm  quality,  not  locate  defects.  This 
scenario-based  method  of  simulation  and 
statistically  driven  testing  has  been  report¬ 
ed  as  30  times  better  than  classical  cover¬ 
age  testing  [7]. 

Page-Jones  dismisses  mathematical 
proofs  of  correctness  because  they  must  be 
based  on  assumptions  [8],  yet  both  testing 
and  correctness  verification  are  done 
against  the  softwares  requirements.  Both 
are  therefore  based  on  the  same  assump¬ 
tions;  incorrect  assumptions  result  in 
incorrect  conclusions.  This  indictment  of 
proofs  of  correctness  must  also  condemn 
testing  for  the  same  reason. 

Table  1 :  TMM  Levels 
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The  clean-room  methodology’s  rigor- 
ous  correctness  verification  approaches 
zero  defects  prior  to  any  execution  [9], 
and  therefore  prior  to  any  testing. 
Correctness  verification  by  mathematical 
proof  seems  better  than  testing  to  answer 
the  question,  “Does  the  software  product 
meet  requirements?” 

Properly  done  test  requirements  analy¬ 
sis,  design,  and  implementation  that  par¬ 
allels  the  same  phases  of  software  develop¬ 
ment  may  help  in  early  defect  detection. 
However,  done  improperly  (as  when 
developers  test  their  own  software),  this 
practice  may  result  in  tests  that  only  test 
the  parts  that  work,  and  in  software  that 
passes  its  tests  but  nevertheless  contains 
defects.  Increasingly  frustrated  users  insist 
that  there  are  serious  defects  in  the  soft¬ 
ware,  while  increasingly  adversarial  devel¬ 
opers  insist  that  the  tests  reveal  no  defects. 

Test  requirements  analysis  done  sepa¬ 
rately  from  software  requirements  analysis 
can  make  successful  testing  impossible.  A 
multi-million  dollar  project  was  only 
given  high-level  requirements,  from  which 
the  software  developers  derived  their  own 
set  of  (often-undocumented)  lower-level 
requirements,  to  which  they  designed  and 
implemented  the  software.  After  the  soft¬ 
ware  had  been  implemented,  a  test  man¬ 
ager  derived  his  own  set  of  lower-level 
requirements,  one  of  which  had  not  even 
been  considered  by  the  developers.  The 
design  and  test  requirements  were  mutual¬ 
ly  exclusive  in  this  area,  so  it  was  impossi¬ 
ble  for  the  software  to  pass  testing.  This 
failure  scrapped  the  entire  project  and 
destroyed  several  careers  [10]. 

Defect  Removal  and 
Prevention 

Test-result-driven  defect  removal  is  detec¬ 
tive  work;  the  maintenance  programmer 
must  identify  and  track  down  the  cause 
within  the  software.  Defect  removal  is  also 
similar  to  archeology,  since  all  previous 
versions  of  the  software,  and  documenta¬ 
tion  of  all  previous  phases  of  the  develop¬ 
ment  may  have  to  be  researched,  if  avail¬ 
able.  Using  testing  to  validate  that  soft¬ 
ware  is  not  defective  [9],  rather  than  to 
identify  and  remove  defects,  moves  their 
removal  from  detection  to  comparative 
analysis  [7]. 

TMM  Level  3  integrates  testing  into 


the  software  lifecycle.  This  includes  testing 
each  procedure  or  module  as  soon  as  pos¬ 
sible  after  each  is  written.  Integration  test¬ 
ing  is  also  done  as  soon  as  possible  after 
the  components  are  integrated.  Neverthe¬ 
less,  the  concept  of  defect  prevention  is 
not  addressed  until  TMM  Level  4,  and 
then  only  as  a  philosophy  for  the  entire 
testing  process. 

Testing  cannot  prevent  the  occurrence 
of  defects;  it  can  only  aid  in  the  prevention 
of  their  recurrence  in  future  components. 
This  is  why  neither  CMM  nor  TMM  dis¬ 
cusses  actual  defect  prevention,  or  more 
accurately,  subsequent  defect  prevention 
until  Level  5.  Waiting  until  one  has 
reached  Level  5  before  trying  to  prevent 
defects  can  be  very  costly,  both  in  terms  of 
correcting  defects  not  prevented  and  in 
lost  business  and  goodwill  from  providing 
defective  software. 

Waiting  until  implementation  to  test  a 
component  for  defects  caused  in  much 
earlier  phases  seems  too  much  of  a  delay; 
yet,  an  emphasis  in  testing  for  defect  pre¬ 
vention  is  exactly  that.  An  ounce  of  pre¬ 
vention  may  be  worth  a  pound  of  cure, 
but  one  cannot  use  a  cure  as  if  it  were  a 
preventative. 

There  are  several  methods  currently 
available  to  accomplish  defect  prevention 
at  earlier  levels  of  maturity  such  as 
Cleanroom  Software  Engineering  [9], 
Zero  Defect  Software  [11],  and  other 
provably  correct  software  methods  [2,  12]. 

Software  Quality  and  Process 
Improvement 

Gene  Krinz,  Mission  Operations’  director 
for  the  NASA  space  shuttle,  is  quoted  as 
saying  about  the  quality  of  the  flight  soft¬ 
ware,  “You  can’t  test  quality  into  the  soft¬ 
ware  [11].”  Clean- room  methods  teach 
that  one  can  neither  test  in  nor  debug  in 
quality  [9]. 

If  quality  was  not  present  in  the 
requirements  analysis,  design,  or  imple¬ 
mentation,  testing  cannot  put  it  there. 
One  of  TMM’s  Level  3  maturity  goals  is 
software  quality  evaluation.  While  many 
quality  attributes  may  be  measured  by 
testing,  and  many  quality  goals  may  be 
linked  to  testing’s  ultimate  objectives, 
most  aspects  of  software  quality  come 
from  the  quality  of  its  design. 

Procedure  coupling  and  cohesion 


[13],  measures  of  object-oriented  design 
quality  such  as  encapsulation,  con- 
nascence,  encumbrance,  class  cohesion, 
type  conformance,  closed  behavior  [8], 
and  other  quantitative  measures  of  soft¬ 
ware  quality,  are  established  in  the  design 
phase.  They  should  be  measured  soon 
after  each  component  is  designed;  do  not 
wait  until  after  implementation  to  meas¬ 
ure  them  with  testing. 

Some  authors  have  suggested  that  ana¬ 
lyzing,  designing,  and  implementing  tests 
in  parallel  with  the  products  to  be  tested 
will  somehow  improve  the  processes  used 
to  develop  those  products.  [3]  However, 
since  the  software  product  testers  should 
be  different  from  those  who  developed  it, 
there  needs  to  be  some  way  for  the  testers 
to  communicate  their  process  improve¬ 
ment  lessons  learned  to  the  developers. 
Testers  and  developers  should  communi¬ 
cate  effectively;  every  developer  should 
also  act  as  a  tester  (but  only  for  compo¬ 
nents  developed  by  others). 

Designing  in  Quality 

One  of  the  maturity  subgoals  of  subse¬ 
quent  defect  prevention  is  establishing  a 
causal  analysis  mechanism  to  identify  the 
root  causes  of  defects.  Already  there  is  evi¬ 
dence  that  most  defects  are  created  in  the 
requirements  analysis  and  design  phases 
[11].  Some  have  put  the  percentage  of 
defects  caused  in  these  two  phases  at  70 
percent  [3]. 

Clear  communication  and  careful 
documentation  are  required  to  prevent 
injecting  defects  during  the  requirements 
analysis  phase.  Requirements  are  charac¬ 
teristically  inconsistent,  incomplete, 
ambiguous,  nonspecific,  duplicate,  and 
inconstant.  Interface  descriptions,  proto¬ 
types,  use  cases,  models,  contracts,  pre- 
and  post-conditions,  etc.  are  all  useful 
tools. 

To  prevent  injecting  defects  during 
the  design  phase,  software  components 
must  never  be  designed  until  a  large  part 
of  their  requirements  have  been  identified 
and  analyzed.  The  design  should  be  thor¬ 
ough,  using  such  things  as  entities  and 
relationships,  data  and  control  flow,  state 
transitions,  algorithms,  etc.  Peer  reviews, 
correctness  proofs  and  verifications,  etc. 
are  good  ways  to  demonstrate  that  a 
design  satisfies  its  requirements. 
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Preventing  the  injection  of  defects  dur¬ 
ing  the  implementation  phase  requires  that 
software  components  never  be  implement¬ 
ed  until  they  have  been  designed.  It  is  far 
too  easy  to  implement  a  software  compo¬ 
nent  while  the  design  is  still  evolving,  some¬ 
times  just  in  the  developers  mind.  Poor 
documentation  and  a  lack  of  structure  in 
the  code  usually  accompany  an  increased 
number  of  defects  per  100  lines  of  code  [3]. 
As  I  mentioned  earlier,  this  applies  even  to 
prototype  and  incremental  software  com¬ 
ponents;  those  experimental  or  partial  fea¬ 
tures  should  be  designed  before  implemen¬ 
tation. 

The  clean-room  method  has  an  excel¬ 
lent  track  record  of  near  defect-free  software 
development,  as  documented  by  the 
Software  Technology  Support  Center,  Hill 
AFB,  Utah,  regardless  of  Daich’s  statements 
to  the  contrary  [3] .  Clean-room  is  compat¬ 
ible  with  CMM  Levels  2  through  5  [9],  and 
can  be  implemented  in  phases  at  all  these 
levels  [6]. 

Conclusion 

It  is  my  dream  that  software  engineering  will 
become  as  much  of  an  engineering  disci¬ 
pline  as  the  others;  users  will  have  just  as 
much  confidence  that  their  software  is  as 
defect  free  as  their  cars,  highway  bridges, 
and  aircraft. 

Testing  should  be  used  to  demonstrate 
the  absence  of  defects,  not  to  identify  defects 
inserted  in  earlier  phases.  It  should  be  used 
to  certify  that  the  software  components 
implement  their  designs,  and  that  these 
designs  satisfy  their  requirements. 

Analyzing  testing  requirements  should  be 
done  in  parallel  with  analyzing  the  software 
components’  requirements.  Tests  should  be 
designed  in  parallel  with  designing  the  com¬ 
ponents.  Test  implementation  should  occur 
in  parallel  with  implementing  the  compo¬ 
nents,  and  developing  integration  tests 
should  be  done  in  parallel  with  integration. 

The  source  of  software  defects  is  a  lack 
of  discipline  in  proper  requirements  analy¬ 
sis,  design,  and  implementation  processes. 
Testing  must  physically  occur  after  imple¬ 
mentation,  so  reliance  on  it  to  detect 
defects  delays  their  correction.  Until  soft¬ 
ware  defects  are  attacked  at  their  source, 
software  will  continue  to  be  developed  as  if 
it  were  an  art  form  rather  than  a  craft,  engi¬ 
neering  discipline,  or  a  science.  ♦ 
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“Let  us  change  our 
traditional  attitude  to 
the  construction  of 
programs.  Instead  of 
imagining  that  our 
main  task  is  to  instruct 
a  computer  what  to  do, 
let  us  concentrate 
rather  on  explaining  to 
human  beings  what  we 
want  a  computer 

to  do.  ” 

—  Donald  Knuth 
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